The invention relates generally to digital audio signal processing and in particular to an improved method and apparatus for use in real-time digital audio processing. Although the invention is described in the environment of a system for large-scale real-time audio processing, such as film sound mixing, special effects processing, and music synthesis, it is applicable in general to other programmable real-time digital signal processing devices.
The basic techniques for digitally synthesizing, processing and recording audio signals are well known. For example, complex waveforms can be digitally generated to stimulate speech and musical instruments or to create totally new sounds. Digital audio, synthesized or converted from analog form, can be processed in highly complex ways to simulate filtering, equalization, etc. Digital audio signals can be recorded and played back. They can be mixed digitally. See for example the articles "Signal Processing Aspects of Computer Music: A Survey" by J. A. Moorer, Proc. I.E.E.E., Vol. 65, No. 8, August, 1977, and the extensive bibliography therein and "Music Synthesis Using Real Time Digital Techniques" by Harold G. Alles, Proc. I.E.E.E., Vol. 68, No. 4, April, 1980. See also "A Professional Digital Audio Mixer" by N. Sakamoto et al., J. Audio Eng. Soc., Vol. 30, No. 1/2, January/February, 1982 and "An Experimental `All-Digital` Studio Mixing Desk" by Richards and Craven, J. Audio Eng. Soc. Vol. 30, No. 3, March, 1982.
One well-known approach to digital audio synthesis and processing is to program a general purpose computer to perform the various digital computations required for particular applications. A serious draw-back to this approach is that a large capacity high speed computer with a powerful instruction set is required for all but the simplest tasks, particularly if multiple simultaneous processing functions or real-time processing are required.
In professional audio applications, the demands on a digital system are particularly severe. For example, to simulate entirely the processing of a modern 32 track mixing console requires a sustained rate on the order of 60 million multiplications per second, a rate that is substantially beyond all but the most powerful of the general-purpose computers.
One known solution to the requirements of high speed calculation coupled with sustained high-bandwidth input/output is to provide a high speed processor to carry out the sequences of digital audio processing operations under the supervision of a slower general purpose computer.
A requirement in all digital signal processors is that the processing parameters (filter coefficients, for example) or the nature of the processing (filtering or equalizing, for example) changes with time (sometimes very quickly). Consequently, some means is required to control the time varying processing not only by means of initial commands implemented in software, hardware or a combination thereof, but also by means of updates to those commands.
In known systems where a high speed processor is supervised by a support computer, the support computer supplies processing commands to the high speed processor. The manner in which commands and command updates are handled presents special problems particularly in systems controlled in real-time.
One such system for digital audio having parallel processors is disclosed in "Microprocessor Mixing and Processing of Digital Audio Signals" by G. W. McNally, J. Audio Eng. Soc., October, 1979, pages 793-803 and in McNally's published U.K. patent application No. 2,017,987. McNally employs two separate computers, a high speed microprogrammable processor and a slower support microprocessor, which are linked by a commonly accessed memory such that in operation the high speed processor can effect processing in dependence upon parameters written into the memory by the slow speed processor. The high speed processor is synchronized with the audio samples during one of the 192 microinstruction cycles between each audio sample, and during that same cycle new parameters can be entered in the memory. The basic nature of the processing itself is determined by the microprogram initially loaded in the high speed processor and which is not changed once loaded. For example, the microprogram may provide for a digital transverse filter to be implemented by means of controlling the arithmetical and logical functions of the CPU within the high speed processor and by employing the filter parameter data in the commonly accessed memory in performing those operations.
Another digital sound processor arrangement in which a high speed processor runs in parallel with a support computer, and in which the high speed processor has a memory directly accessible by the support computer is the 4C system designed as a musical performance device at IRCAM by Pepino di Giugno and described in "The 4C Machine" by Moorer et al., Computer Music Journal, Vol. 3, No. 3, Fall, 1979, pages 16-24 and in "Synthesizers I Have Known and Loved, by J. A. Moorer, Computer Music Journal, Vol. 5, No. 1, Spring, 1981, pages 4-12.
The 4C System's high speed processor (the 4C "Machine"), is more elaborate than in the McNally system, providing seven PROM's (programmable read only memories) accessible to the support computer allowing not only changes to the parameter data but also changes to the machine programming. Thus, the support computer can also change the nature of the high speed processing.
Both the McNally system as disclosed and the 4C System suffer from the problem that both place a substantial burden on the support computer to issue commands for immediate execution by the high speed processor and there is no means for synchronizing the exact arrival time of the data update commands with a particular sample number. This is disadvantageous because audible clicks or pops can occur for example, from the non-simultaneous onset of processing associated with a sound beginning at a particular sample time. Also, in both systems the processor clock period has a random phase with respect to the support computer instruction cycle. Consequently, there is an unpredictable time delay between the generation of a data update by the support computer and its execution by the processor. This is a further disadvantage that can cause results such as perceptual stereo image shifts and relative microshifts within a musical note.
A large scale digital synthesizer having certain general signal processing capabilities is described in "A General-Purpose Digital Synthesizer" by Peter R. Samson, J. Audio Eng. Soc., March, 1980, pages 106-112, and in the above cited articles "Signal Processing Aspects of Computer Music: A Survey" (at pages 1134-1135, "The Systems Concepts Digital Synthesizer") and "Synthesizers I Have Known and Loved". The Systems Concepts device has a fixed menu of functional modules: up to 256 "generators" and 128 "modifiers", each of which having a control memory, written by a CPU, that determines its function and holds its running values and parameters. A 28-command FIFO (first-in, first-out memory) stores all data updates to the control memories in the form of "linger" commands from the support computer that contain timing information. Thus, the system assures that updates will occur during a specific sample and lessens the burden on the support computer by allowing it to fill the command FIFO ahead of time. The latter feature is particularly useful because update commands tend to occur in bursts of extremely high bandwidth. However, this arrangement does not allow for adjustment of parameters in real-time, since each command is held for execution until the sample time tagged to it before the next command is examined. Hence, a command insert will not take effect until some time in the future, depending on what commands are already in the FIFO.
Another digital audio processor in the form of a parallel system is described in the above cited article "An Experimental `All-Digital` Studio Mixing Desk" by Richards and Craven. In the system described real-time digital processing hardware is supported by a central control processor. The digital processing hardware provides certain dedicated mixing and equalization functions; it is not programmable as are the McNally and 4C Systems.
Thus, several systems are known having a high speed digital audio processor running in parallel with a slower support computer, in which the high speed processor has a memory accessible to the support computer into which it places processing commands. The high speed processors range from dedicated hardware to programmable general purpose hardware. In the McNally and 4C Systems, the processing commands are untimed. That is, they cannot be associated with specific sample times and are executed immediately subject to the timing uncertainties discussed above. In the Systems Concepts system, the processing commands are stored in a FIFO and are timed such that they are always associated with specific sample times.
The above cited article by Moorer, "Synthesizers I have known and Loved" discusses generally the problems in configuring a command structure, including those inherent in "timed and untimed" systems, such as those just mentioned. Also discussed is the problem of merging multiple sets of commands, as occurs, for example, in synthesizing noninteracting musical notes, each note being composed of a number of simultaneously emerging envelopes. In the known systems, this merge is accomplished in the support computer by software that interleaves the data into a single stream which is then written into the high speed processor. In the case of the Systems Concepts' embedded timing information command stream, keyboard control or other real-time asynchronous control events essentially would be ruled out because the new data would have to be merged into the already existing command stream. This is not possible for the commands that are already in the FIFO (new commands go to the end of the list).
Thus, although the timed command approach in the Systems Concept device overcomes certain of the inherent problems in the untimed command approach of the 4C and McNally Systems, it is apparent that it is not practical for a system that must respond to unplanned asynchronous real-time user controls.
In the conclusion of the said "Synthesizers I Have Known and Loved" article it is suggested that if real-time control by a user is desired that either a secondary command update path be provided or that timing information on commands be eliminated as in the 4C system. The secondary path suggestion has the disadvantage that the complexity resulting from a second command stream is created. The disadvantages of the 4C system approach have been discussed. The article also suggests as a solution to the merge problem a microprogrammed processor in which sequentially running programs can read independently from separate command data streams. A significant disadvantage of such an approach is that a very large number of parallel processes would be required in a practical system.
Thus, the prior art has failed to provide an optimized command structure for digital processing systems of the type where a high speed processor has a slower support computer. Particularly, the prior art does not solve the problem of a practical real-time processing system with real-time control such that command updates of known and unknown timing are simultaneously present.
A timed command may be defined as a command that is to take place at a specific and known sample number (or between two specific and known sample numbers). An untimed command is one for which the sample number is not known beforehand.
More specifically the prior art fails to provide a command organization that can accommodate not only a known list of timed commands in addition to asynchronous untimed commands but also the merging of a further known list of timed commands that immediately results from many untimed commands. For example, if the processor is being played like an organ, there may be a known score running, which has many timed commands. If an organ key is pressed, the attack portion of the note requires a precisely timed sequence of commands, the timing of which is known as soon as the key is pressed, but it was not known until the key was pressed.